Non-rigid stereo vision camera system

ABSTRACT

A long-baseline and long depth-range stereo vision system is provided that is suitable for use in non-rigid assemblies where relative motion between two or more cameras of the system does not degrade estimates of a depth map. The stereo vision system may include a processor that tracks camera parameters as a function of time to rectify images from the cameras even during fast and slow perturbations to camera positions. Factory calibration of the system is not needed, and manual calibration during regular operation is not needed, thus simplifying manufacturing of the system.

CROSS REFERENCE TO RELATED APPLICATIONS

The present application is a continuation of International ApplicationNo. PCT/US2021/12294 filed Jan. 6, 2021, entitled “Non-Rigid StereoVision Camera System,” which claims the benefit of priority of U.S.Provisional Application No. 62/964,148 filed Jan. 22, 2020, entitled“Untethered Stereo Vision Camera System.” The entire contents of theseapplications is incorporated by reference herein.

TECHNICAL FIELD

The technology of the present invention relates to stereo visionsystems. In particular, the present technology relates to a stereovision system (e.g., a stereo camera system) having characteristics forimproved operation in the presence of fast and slow mechanicalperturbations, and to methods for autocalibration of the system.

RELATED ART

Stereo camera systems may be used to acquire three-dimensionalinformation about objects in the field of view by triangulationtechniques.

Conventional stereo camera systems may use rigid mounting members to fixthe position of cameras with respect to each other. For example, U.S.Pat. No. 6,392,688B1 discloses the use of a thick metal plate as a rigidmounting member for direct attachment of cameras and their associatedCMOS sensors and lenses to mechanically stabilize the relative positionof the cameras. The thickness of the plate shown in FIG. 5 of thisdocument is approximately one-fifth of the distance between the cameras,which makes such a structure very heavy, which may not be suitable foruse in a system with wide-baseline stereo cameras. For example, a stereocamera system with a 1-m baseline between cameras would need a20-cm-thick metal plate, which would weigh 108 kg if made of aluminum (1m×0.2 m×0.2 m). The terms “wide baseline” and “long baseline” may beused interchangeably herein.

A problem with such existing stereo camera systems is that individualcamera modules (sometimes referred herein to as “camera sensors” orsimply “cameras”) may shift and/or rotate with respect to each otherover time, which may make initial calibration parameters becomeinaccurate over time. It is desirable for the fields of view of thecamera modules of a stereo camera system not to move relative to eachother in order for the stereo camera system to remain properlycalibrated. For some stereo camera systems, a relative cameraorientation rotation of only 0.05 degrees can ruin the calibration ofthe system. Moreover, in a conventional stereo camera system, there ispotential for movement of, e.g., lens holders relative to circuit boardsand/or other camera components, for relative pointing between cameramodules to change, and also for positions of the circuit boards and theframe itself to change over time. The multiple interfaces betweencomponents in a conventional stereo camera system make it likely thatvibration, shock, and even thermal expansion between components willcause the field of view of the cameras in a stereo camera system toshift over time. The relative movement of the camera components mayinvalidate the initial factory calibration of the system, which may makestereo vision data obtained by the system unreliable.

Such calibration problems have not been addressed or even recognized inthe prior art because prior-art stereo vision systems typically haveeither been laboratory systems, which generally are not subjected toshocks or vibrations, or have been used in situations such asshort-range indoor robotics, which generally are not situations wherehighly accurate calibration is regarded as being critical. The inherentsusceptibility of conventional stereo vision systems to losingcalibration has therefore not been recognized as a particular problem tobe solved, because conventional systems have been limited to shortbaseline lengths and utilize large mechanical stiffeners.

Conventional automatic calibration methods typically fall into twocategories: sparse keypoint approaches and vanishing point approaches.Keypoints are points of interest that may be easily recognized inimages, such as corners or edges, and often may have associated imagedescriptors. For example, a scale invariant feature transform (SIFT)method developed by David Lowe is an example of an algorithm for findingkeypoints and providing a descriptor that is invariant to translations,rotations, and scaling transformations. Vanishing points are used torepresent a point at infinite distance. Vanishing points are sensitiveto camera-module orientation but not to relative translation between thecamera modules, and can be used to align the orientation of a pair ofstereo camera modules.

U.S. Pat. No. 8,797,387B2 and U.S. Ser. No. 10/097,812B2 disclosemethods for automatically calibrating stereo vision systems using sparsekeypoint approaches. The method in U.S. Pat. No. 8,797,387B2 extractskeypoints, matches keypoints in images from first and second cameras,and determines a camera calibration based on a single-valuedecomposition analysis of a vertical error. The method in U.S. Ser. No.10/097,812B2 is similar to that of U.S. Pat. No. 8,797,387B2 but trackskeypoints over multiple image frames and uses a structure-from-motiontechnique to apply a three-dimensional constraint. Sparse keypointapproaches can fail when keypoints are incorrectly matched betweenimages from the first and second cameras, which can happen in imagescenes with repeating structures (e.g., picket fences, building windows,etc.). Furthermore, sub-pixel accuracy of feature points may be requiredto obtain camera parameters with sufficient accuracy for long-baselineand long-range operation, which is often not possible with smooth orrounded features, or with slightly defocused or blurred images.Conventional structure-from-motion calculations generally are not fastenough to compensate for calibration errors from frame to frame, andthus typically are appropriate for slow perturbations of short-baselinestereo vision systems. Another problem with sparse keypoint approachesis the sparse sampling of an image, which does not provide enoughinformation to determine the intrinsic or extrinsic camera parametersaccurately. For example, images of typical road scenes may have mostfeatures clustered in the center of the image away from the sky and awayfrom the textureless road surface, and therefore sampling occursgenerally in the center of the image. This clustered sampling does notprovide enough information to determine with high accuracy the relativeorientation of the stereo cameras (an extrinsic camera parameter), muchless lens distortion coefficients (an intrinsic camera parameter), whichare sensitive to distortions at the edges of the image. In other words,these conventional techniques do not compensate for camera parametersthat may be important for proper calibration of a stereo vision system.

Japanese patent JP2008-509619A discloses a method for automaticallycalibrating stereo vision equipment by searching for a vanishing pointand correcting for pitch and yaw errors. The method does not correct forroll errors or for a relative translation vector between the equipment'stwo cameras. Furthermore, the method requires straight and reliable roadpaint markings, which may not be available or may be obstructed by snowor faded by wear or sun exposition, thus limiting angular accuracy ofthe equipment's calibration.

A 2018 conference paper entitled “Flexible Stereo: Constrained,Non-rigid, Wide-baseline Stereo Vision for Fixed-Wing Aerial Platforms,”by authors T. Hinzmann, T. Taubner, and R. Siegwart, discloses a methodfor stereo vision using components mounted on the wings of a modelairplane. Because the wings are flexible and may move, the authorsdeveloped a system to compensate for relative camera motion. The systemrequires the use of inertial measurement units (IMUs) attached directlyto stereo cameras to measure acceleration and angular rates of thecameras. The system then computes the relative orientation and positionof the cameras using an extended Kalman filter. One disadvantage of thissystem is that it requires IMU hardware and cannot correct for cameraposition from a video stream alone. Another disadvantage is that theIMUs can be used to compensate for fast perturbations but not for slowperturbation (e.g., a slow drift of positions of the cameras), whichmeans that the cameras may need to be manually calibrated on a frequentbasis (e.g., daily).

SUMMARY

The calibration problems described above may be exacerbated inlong-baseline stereo vision systems, where the distance between cameramodules is, e.g., greater than approximately 20 cm, and in systems whereit is not possible to connect two or more camera modules with heavystructural beams or mounting members. For styling and for an optimalviewing vantage point, it is sometimes desirable to mount independentcameras on structures that are non-rigid (e.g., a flexible structure).For example, mounting a pair of stereo cameras in a vehicle'sheadlights, at upper corners of a windshield of a vehicle, or onside-view mirrors of a vehicle, would be advantageous as such locationswould be convenient for establishing a long baseline for a stereo visionsystem in a vehicle (e.g., a car, a truck, a bus, etc.). However, theselocations are not sufficiently rigid to maintain calibration over hoursor days, much less a 15-year lifetime of a typical vehicle. In somecases, it may not be realistically possible to add rigid mechanicalsupport structures between the headlights, the upper corners of thewindshield, and the side-view mirrors, because a shortest path for thesesupport structures may be blocked by an engine block, may block adriver's view, and may be blocked by the driver himself/herself,respectively. Furthermore, even if supporting structures could be added,a thickness and weight of such structures required to provide adequatestructural support may be impractical. That is, because a beam'sdeflection displacement scales as the cube of the beam's length, thelonger the beam the greater the beam may displace for the same amount offorce applied to the beam. For example, for the same end force, a 2-mbaseline stereo vision system will displace 1000 times more than a 20-cmbaseline stereo vision system. This scaling has resulted in commerciallyavailable stereo vision systems to be physically limited to baselines ofless than 20 cm.

There is a need for a stereo vision system that may operate in highvibrational and shock environments, that may be mounted on non-rigidstructures, that may support long-baseline and long-range stereo vision,and that may operate with high accuracy.

Automatic calibration technology suitable for long-baseline andnon-rigid structures has not been achieved prior to the technologydisclosed herein for at least four reasons: (1) magnitude: by physics,beam displacements increase as the cube of beam length (see discussionabove) and therefore the magnitude of relative camera displacementsincreases with beam length. The additional structural support to ensurerigidity and prevent camera displacements may not be realisticallyachievable; (2) bandwidth: a calibration speed of an autocalibrationsystem should increase with increasing baseline length to account forboth slow and fast variations. The processor hardware required toachieve the needed computational speed may be costly and may addunwanted complexity to the system; (3) accuracy: a calibration accuracyneeded for long-baseline systems may be much greater than forshort-baseline systems, because greater angular accuracies are requiredto estimate targets at longer distances; and (4) completeness: whereasconventional techniques for automatic calibration of stereo cameras maycompensate for a few camera parameters (e.g., the relative orientationof the stereo cameras), a full calibration would require compensatingfor all extrinsic and intrinsic camera parameters. Prior to thetechnology presented herein, a system that addresses these four reasonshas not yet been achieved.

The inventive technology described herein and claimed in the claims isdirected to a stereo vision system in which no rigid mounting member isrequired. Vision may be achieved via a plurality of vision sensors(e.g., camera modules). In some aspects of the present technology, firstand second camera modules (e.g., camera sensors) may be placed on orattached to structures that may flex, shift, bend, or/or move. As notedelsewhere herein, the terms camera sensor, camera module, and camera maybe used interchangeably. A stereo camera system according to aspects ofthe present technology may be comprised of a processor configured toperform active tracking and compensation for movement of a first camerarelative to a second camera, to provide accurate depth maps over allmotion frequencies from slow material deformations due to, e.g.,temperature changes, to fast vibrational motions due to, e.g., road orengine noise. Because there is no requirement for rigid mounting ofcameras, aspects of the present technology may enable wide-baselinestereo configurations for long-range measurements without requiringperiodic manual calibration.

According to an aspect of the present technology, a stereo vision systemis provided. The stereo vision system may be comprised of: a firstcamera sensor configured to sense first reflected energy of a firstimage and to generate first sensor signals based on the first reflectedenergy; a second camera sensor configured to sense second reflectedenergy of a second image and generate second sensor signals based on thesecond reflected energy; at least one processor configured to receivethe first sensor signals from the first camera sensor and the secondsensor signals from the second camera sensor, and to producethree-dimensional (3D) data from the first and second sensor signals.The at least one processor may be configured to: generate rectifiedstereo images from the first and second sensor signals utilizing stereocalibration parameters, perform a stereo matching on the rectifiedimages, and perform an automatic system calibration using data from aplurality of stereo images obtained by the first and second camerasensors. The automatic system calibration may be based on minimizing acost function. In some embodiments of this aspect, the first reflectedenergy and/or the second reflected energy may be comprised of lightenergy or photons originating from at least one electronic device (e.g.,headlight(s), streetlight(s), laser(s), etc.). In some embodiments ofthis aspect, the first reflected energy and/or the second reflectedenergy may be comprised of solar light energy or photons originatingfrom the sun. In some embodiments of this aspect, the first reflectedenergy and/or the second reflected energy may be comprised of acombination of solar light energy and light energy originating from anelectronic device.

According to another aspect of the present technology, acomputer-implemented method for detection of an object in an image isprovided. The method, which may be performed by one or moreprocessor(s), may be comprised of: identifying connected componentregions on the object by color; determining an aspect ratio of each ofthe connected component regions; determining a distance between theconnected component regions; and identifying the object to be a knownobject based on the aspect ratios and the distance between the connectedcomponent regions. Any one or any combination of: a presence, alocation, and a size of the known object may be determined from theimage.

According to another aspect of the present technology, acomputer-implemented calibration method to calibrate a stereo visionsystem is provided. The method may be performed by one or moreprocessor(s). The stereo vision system may include a first camera sensorconfigured to sense first reflected energy of a first image and togenerate first sensor signals based on the first reflected energy and asecond camera sensor configured to sense second reflected energy of asecond image and generate second sensor signals based on the secondreflected energy. The method may be comprised of: generating stereoimages from the first and second sensor signals; rectifying the stereoimages using stereo calibration parameters, to produce rectified stereoimages; performing a stereo matching on the rectified stereo images; andperforming an automatic system calibration using a result of the stereomatching. The automatic system calibration may be based on minimizing acost function. In some embodiments of this aspect, the first reflectedenergy and/or the second reflected energy may be comprised of lightenergy or photons originating from at least one electronic device (e.g.,headlight(s), streetlight(s), laser(s), etc.). In some embodiments ofthis aspect, the first reflected energy and/or the second reflectedenergy may be comprised of solar light energy or photons originatingfrom the sun. In some embodiments of this aspect, the first reflectedenergy and/or the second reflected energy may be comprised of acombination of solar light energy and light energy originating from anelectronic device.

According to another aspect of the present technology, a non-transitorycomputer readable medium is provided in which is storedcomputer-executable code that, when executed by one or moreprocessor(s), may cause the one or more processor(s) to calibrate astereo vision system that may include a first camera sensor configuredto sense first reflected energy of a first image and to generate firstsensor signals based on the first reflected energy and a second camerasensor configured to sense second reflected energy of a second image andgenerate second sensor signals based on the second reflected energy. Themethod may be comprised of: generating stereo images from the first andsecond sensor signals; rectifying the stereo images using stereocalibration parameters, to produce rectified stereo images; performing astereo matching on the rectified stereo images; and performing anautomatic system calibration using a result of the stereo matching. Theautomatic system calibration may be based on minimizing a cost function.In some embodiments of this aspect, the first reflected energy and/orthe second reflected energy may be comprised of light energy or photonsoriginating from at least one electronic device (e.g., headlight(s),streetlight(s), laser(s), etc.). In some embodiments of this aspect, thefirst reflected energy and/or the second reflected energy may becomprised of solar light energy or photons originating from the sun. Insome embodiments of this aspect, the first reflected energy and/or thesecond reflected energy may be comprised of a combination of solar lightenergy and light energy originating from an electronic device.

BRIEF DESCRIPTION OF DRAWINGS

FIGS. 1A-1C show three different locations for mounting cameras in or ona car without using or requiring large and heavy structural supportmaterial between the cameras, according to some embodiments of thepresent technology.

FIGS. 2A-2C show translucent or partially transparent views of the carsof FIGS. 1A-1C, respectively, according to some embodiments of thepresent technology.

FIGS. 3A-3F show view of forward-looking stereo-vision camera modulesmounted in a car at various locations, according to some embodiments ofthe present technology.

FIG. 4 shows a view of a forward-looking stereo-vision camera sensorsmounted on a truck, according to some embodiments of the presenttechnology.

FIGS. 5A-5C show a perspective view, a plan view, and a frontelevational view of a side- and back-looking camera modules of a stereovision system, with the camera modules mounted on a side-view mirror ofa truck, according to some embodiments of the present technology.

FIG. 6 shows a schematic view of cameras of a stereo vision systemmounted on a traffic light, according to some embodiments of the presenttechnology.

FIG. 7 shows a schematic view of cameras of a stereo vision systemmounted on a lamp post, according to some embodiments of the presenttechnology.

FIG. 8 shows a schematic view of cameras of a stereo vision systemmounted on a construction crane, according to some embodiments of thepresent technology.

FIG. 9 shows a schematic view of cameras of a stereo vision systemmounted on a fixed-wing aircraft, according to some embodiments of thepresent technology.

FIGS. 10A and 10B show a front perspective view and a rear perspectiveview, respectively, of camera arrangements of a stereo vision systemmounted on an off-road or all-terrain vehicle, according to someembodiments of the present technology.

FIG. 11 shows a schematic view of cameras of two stereo vision systemsmounted on a combine harvester, according to some embodiments of thepresent technology.

FIGS. 12A, 12B, and 12C show a side elevational view, a frontelevational view, and a plan view of cameras of a stereo vision systemmounted on an automated forklift, according to some embodiments of thepresent technology.

FIG. 13 shows a block diagram of a stereo vision system, according tosome embodiments of the present technology.

FIG. 14 shows a block diagram of a processing component of the system ofFIG. 13, according to some embodiments of the present technology.

FIG. 15A shows a flow diagram of a rectification algorithm utilized insome embodiments of the present technology; and FIG. 15B shows geometricdetails of a rectification procedure performed by the algorithm,according to some embodiments of the present technology.

FIG. 16 shows a block diagram of an autocalibration engine of theprocessing component of FIG. 14, according to some embodiments of thepresent technology.

FIG. 17A shows a block diagram of a calibration engine of theautocalibration engine of FIG. 16, according to some embodiments of thepresent technology.

FIG. 17B shows a flow diagram of a fast optimization method, accordingto some embodiments of the present technology.

FIG. 17C-1 shows a flow diagram of a medium optimization method,according to some embodiments of the present technology.

FIG. 17C-2 shows a flow diagram of an angle-search process of the mediumoptimization method of FIG. 17C-1, according to some embodiments of thepresent technology.

FIGS. 17D-1 through 17D-4 show flow diagrams of a slow optimizationmethod, according to some embodiments of the present technology.

FIG. 18 shows a flow diagram of a procedure of a stereo imagestabilization engine of the autocalibration engine of FIG. 16, accordingto some embodiments of the present technology.

FIG. 19A shows a block diagram of a procedure of an absolute rangecalibration engine of the autocalibration engine of FIG. 16, in whichthe procedure has a non-negative disparity constraint, according to someembodiments of the present technology; and FIG. 19B is a chart showinghow a number of pixels with negative disparity can vary as a function ofyaw, according to some embodiments of the present technology.

FIG. 20A shows a flow diagram of a procedure of an absolute rangecalibration engine of the autocalibration engine of FIG. 16, in whichthe procedure is for an object with known dimensions, according to someembodiments of the present technology; and FIG. 20B shows an example ofimaging of an object with known dimensions, according to someembodiments of the present technology.

FIG. 21A shows a flow diagram of a procedure of an absolute rangecalibration engine of the autocalibration engine of FIG. 16, in whichthe procedure utilizes vehicle odometry, according to some embodimentsof the present technology; and FIG. 21B is a chart showing how disparitycan vary as a function of range, according to some embodiments of thepresent technology.

FIG. 22 shows a flow diagram of a procedure of a calibration manager ofthe autocalibration engine of FIG. 16, according to some embodiments ofthe present technology.

FIG. 23 shows a time series input diagram of the calibration manager ofFIG. 22, according to some embodiments of the present technology.

FIG. 24 shows a flow diagram of a procedure of a stop sign detector,according to some embodiments of the present technology.

FIG. 25 shows a diagram illustrating a procedure of a stereocorrespondence engine of the processing component of FIG. 14, accordingto some embodiments of the present technology.

DETAILED DESCRIPTION

The inventors have developed image processing techniques that may enablea stereo vision system to produce depth maps with high quality even inthe presence of vibrations and even when a mounting structure forcameras of the system is not rigid (e.g., the mounting structure may bedeformed easily from external forces). The inventors have recognizedthat a camera autocalibration method that may compensate for both fastand slow perturbations may enable long-baseline (also referred to hereinas “wide-baseline”) stereo vision systems to be used on dynamicplatforms, such as robots, cars, trucks, light posts, constructioncranes, aircraft, etc.

For vehicles such as automobiles and trucks, there are mountinglocations that may be preferred or even ideal for aesthetic design, formanufacturing, and/or for providing optimal vantage points for sensorsor cameras of a stereo vision system; however, these mounting locationsmay not have sufficient rigidity or stability to support a strictangular tolerance of cameras used in conventional stereo vision systems.Perturbing a relative pointing direction of stereo cameras by only 0.05degrees of some conventional stereo vision systems can result in depthmaps that may be ruined because they provide unreliable range estimates.

FIGS. 1A-1C show three different locations for mounting cameras in or ona vehicle without using or requiring large and heavy structural supportmaterial between the cameras, according to some embodiments of thepresent technology.

FIG. 1A shows a mounting configuration of a stereo vision systemaccording to some embodiments of the present technology, in which a leftcamera 100 and a right camera 102 of the system may be mounted at a topportion of a front windshield of a vehicle. This location may bedesirable because the cameras 100, 102 are internal to a cabin of thevehicle and therefore may be protected from dust and rain. For example,windshield wipers may be used to clear debris (e.g., rain, snow, leaves,etc.) away from a field of view of the cameras 100, 102. In someembodiments, the right camera 100 and the left camera 102 may be mountedasymmetrically and may be arranged to be within a cleaning zone of thevehicle's windshield wipers. In some embodiments, a typical baselinewidth between the right camera 100 and the left camera 102 may beapproximately 80 cm, which is a width that, with conventional stereovision systems, would require an impractically thick, heavy, rigid,structural member to connect the two cameras 100, 102. As will beappreciated, such a structural member could obstruct a human driver'svision. Conventional autonomous test vehicles that have stereo cameraswith a baseline that is greater than 50 cm typically use bulky slottedaluminum members with 45 mm×45 mm or 90 mm×90 mm profiles to provideenough rigidity to maintain camera alignment over a day or sometimeslonger.

FIG. 1B shows a mounting configuration of a stereo vision systemaccording to some embodiments of the present technology, in which a leftcamera 104 and a right camera 106 of the system may be mounted on a roofportion of a vehicle. The roof portion may provide a high vantage point,which may allow the vision system to see over objects in front of thevehicle and to be far from splash and spray from wet pavement and roaddebris. In some embodiments, the camera 104, 106 may be mounted onto aroof headliner without using extra structural material to stabilize thecameras 104, 106. Furthermore, in some embodiments, camera enclosuresused with the camera 104, 106 may contain one or more cleaning devices(e.g., heater elements configured to remove frost and/or to defog, asprayer for liquid and/or air configured to remove solid debris, etc.).In some embodiments, a typical baseline width between the left camera104 and right camera 106 may be 1.25 m. In some implementations, thecameras modules 104, 106 on the roof portion of the vehicle may havelittle or no effect on the vehicle's styling. For example, in someembodiments, the cameras 104, 106 may be placed seamlessly in line witha roof rack of the vehicle.

FIG. 1C shows a mounting configuration of a stereo vision systemaccording to some embodiments of the present technology, in which a leftcamera 108 and right camera 110 of the system may be mounted onside-view mirrors of a vehicle. The side-view mirrors may offer a widestbaseline, typically 2.1 m, for a forward-looking stereo vision system,but may experience more perturbations than the mounting configurationsshown in FIGS. 1A and 1B. For typical vehicles, the side-view mirrorsare at the extremities of the vehicle. Typical side-view mirrors aremounted to an articulated joint and are light structures that aresensitive to internal vibration sources (e.g., the vehicle's engine) aswell as external perturbations (e.g., from wind, road vibration, etc.).Cameras mounted on side-view mirrors generally experience fastperturbations as well as slow perturbations.

FIGS. 2A-2C show translucent or partially transparent views of thevehicles of FIGS. 1A-1C, respectively, according to some embodiments ofthe present technology. In some embodiments, wires 200, 202, 204, 206,208, 210 may connect the cameras 100, 102, 104, 106, 108, 110 toelectronic control units 212, 214, 216. The wires 200, 202, 204, 206,208, 210 may be configured to transmit data to the electronic controlunits 212, 214, 216 and supplying power to the cameras 100, 102, 104,106, 108, 110.

FIG. 3A through FIG. 12 show various embodiments of stereo visionsystems on different platforms that may move or flex, that may besubject to vibrations and shocks, and that may be inconvenient or costlyto calibrate manually, and/or that may need to operate for years withoutmanual calibration and maintenance. In FIG. 3A through FIG. 12, camerasof the systems may be shown mounted on the platforms but electroniccontrol units and wiring may not be shown as they may be mounted insidea chassis of the platforms or inside the platforms themselves, accordingto some embodiments of the present technology.

FIGS. 3A-3F show camera modules C1, C2 for a forward-looking stereovision system mounted in or on a car at various locations, according tosome embodiments of the present technology. These locations typically donot provide optical rigidity to ensure alignment of stereo visioncameras over the car's lifetime nor even over temperature rangesexperienced by typical cars during, e.g., seasonal changes and/ortemperature swings from nighttime to daytime. FIG. 3A shows the camerasC1, C2 mounted on side-view mirrors. FIG. 3B shows the cameras C1, C2mounted on a roof. FIG. 3C shows the cameras C1, C2 mounted behind thecar's windshield glass. FIG. 3D shows the cameras C1, C2 mounted in thecar's grille. FIG. 3E shows the cameras C1, C2 mounted in the car'sheadlight housings. FIG. 3F shows the cameras C1, C2 mounted in thecar's fog-light housings.

FIG. 4 depicts camera modules C1, C2 of a forward-looking stereo visionsystem according to some embodiments of the present technology, with thecamera modules C1, C2 mounted on side-view mirrors of a truck. The truckmay have a wide baseline length of approximately 3 m from a leftside-view to a right side-view mirror, which may enable very long-rangestereopsis. In some embodiments, the camera modules C1, C2 may becomprised of CMOS sensors.

FIGS. 5A-5C show an arrangement for a side- and back-looking stereovision system, according to some embodiments of the present technology,with camera modules 500, 502 shown to be mounted on a right side-viewmirror of a truck in FIG. 5A. FIG. 5A is an isometric or perspectiveview, FIG. 5B is a top or plan view, and FIG. 5C is a front elevationalview of the truck. In some embodiments, a duplicate stereo vision systemmay be also mounted on a left side-view mirror of the truck, to senseobjects on the truck's left side. The duplicate stereo vision system isnot shown in the drawings, to avoid unnecessary clutter.

The two camera modules 500, 502 of the stereo vision system may bemounted vertically and may be spaced apart by approximately 1 m. As willbe appreciated, stereo matching may be performed to match images fromcameras oriented any angle relative to each other. For example, stereomatching of images from cameras arranged vertically relative to eachother may correlate horizontal lines or rows of pixels of the images,stereo matching of images from cameras arranged horizontally relative toeach other may correlate vertical lines or columns of pixels of theimages, etc. A depth range of a stereo vision system may be proportionalto a distance between cameras of the system. For the system shown inFIGS. 5A-5C, the system may have a horizontal field of view 504 of 135degrees, as depicted in FIG. 5B, and may have a vertical field of view506 of 90 degrees, as depicted in FIG. 5C. As will be appreciated, inother embodiments of the present technology other fields of view arepossible via, e.g., different camera lenses (e.g., standard lenses,different types of wide-angle lenses, etc.). A side- and back-lookingstereo vision configuration, such as that of FIGS. 5A-5C, may beparticularly useful for sensing and thus avoiding pedestrians, sensingand thus avoiding cyclists, and sensing vehicle cut throughs.Conventional stereo vision systems typically have very poor depthranges, especially wide field-of-view systems, because in such systemsthe ability to estimate angles accurately degrades as each pixel ismapped to a larger range of angles. As will be appreciated, adepth-range resolution of a stereo vision system may be proportional tothe system's field-of-view. For a wide-baseline stereo vision system itmay be advantageous to use wide field-of-view cameras because thesystem's depth-range resolution may be inversely proportional to abaseline length of the system's stereo cameras. Therefore, embodimentsof the present technology may enable long-range, surround-viewthree-dimensional (“3D”) sensing of objects by enabling longstereo-vision baselines (e.g., 1 m or greater) in comparison with onlyabout 5-cm to 10-cm baselines in conventional stereo imaging systems.

FIG. 6 shows two cameras 600, 606 of a stereo vision system mounted on atraffic light pole 602, according to some embodiments of the presenttechnology. The two cameras 600, 606 may be separated by a long baselinelimited by a length of a boom 602 a of the traffic light pole 602. Sometraffic light poles may support stereo-vision baseline lengths ofapproximately 5 m, which may allow for extremely long-range depthestimation (e.g., in excess of 300 m), which may enable measurement of aqueue length of vehicles in a traffic queue for intelligent switching oftraffic lights 604. As will be appreciated, booms of traffic light polesmay flex (e.g., with high winds, movement of objects dangling from thebooms, road vibrations from passage of large trucks nearby, etc.). Suchflexing may be experienced by the cameras 600, 606, and such flexing maybe magnified by long poles 608 holding the cameras 600, 606. Thus, along baseline length may cause the two cameras 600, 606 to misalignquickly in a typical operating environment of the traffic light pole602.

FIG. 7 shows two cameras 702, 704 of a stereo vision system mounted on alamp post 700, according to some embodiments of the present technology.The two cameras 702, 704 may be mounted vertically and may be pointedslightly downward to view a region of interest (e.g., a pedestriancrosswalk). For example, an upper edge 706 of the field of view of thecamera 702 may be deviated from a horizontal line by less than about 30degrees (e.g., 20 degrees, 10 degrees, 5 degrees). A baseline distancebetween the two cameras 702, 704 may be several meters or more (e.g., 3m, 3.5 m, 4 m, 4.5 m, etc.). A similar vertical configuration of camerasmay be mounted on sign poles, telephone/utility poles, and the like.

FIG. 8 shows cameras 804, 806 of a stereo vision system mounted on aconstruction crane 800, according to some embodiments of the presenttechnology. The cameras 804, 806 may be mounted on a trolley 802 andarranged to have a downward field of view to, e.g., monitor a payload808 and a surrounding construction environment nearby and underneath thepayload 808. A typical field of view 810 may be about 90 degrees toabout 120 degrees and may extend in the direction parallel or nearlyparallel to a boom of the crane 800. A distance between the cameras 804,806 may be about 2 m to 5 m, depending on the desired minimum depthrange and depth-range resolution. A boom length of approximately 50 mand a crane height of approximately 200 m may make a wide-baseline,long-range stereo vision system an attractive solution for monitoring anenvironment of construction cranes, especially when construction shocksand vibrations may make it difficult to maintain a constant a relativeposition between the cameras 804, 806.

FIG. 9 shows two cameras 902, 904 of a stereo vision system mounted on afixed-wing aircraft 900, according to some embodiments of the presenttechnology. The aircraft 900 may be, e.g., a crop duster, an unmannedaerial vehicle (UAV), a quad- or hexa-copter, an airplane, and the like.In some embodiments, the cameras 902, 904 may be mounted on an undersideof a wing of the aircraft 900 and pointed downward to view a groundregion 906. In some embodiments, the two cameras 902, 904 may be pointedin a forward direction or in a backward direction. Because the wings mayflex or move during operation of the aircraft 900, the cameras 902, 904may oscillate in position back and forth with respect to each other andstereopsis may not be possible using regular stereo vision techniques.

FIGS. 10A and 10B show a front perspective view and a rear perspectiveview, respectively, of camera arrangements of a stereo vision systemmounted on an off-road vehicle 1000, according to some embodiments ofthe present technology. The stereo vision system may have asurround-view configuration comprised of four stereo-vision camerapairs: a front stereo-vision camera pair 1002, a left stereo-visioncamera pair 1004, a back stereo-vision camera pair 1006, and a rightstereo-vision camera pair 1008. The four stereo-vision camera pairs1002, 1004, 106, 1008 may provide four horizontal fields of view of atleast 90 degrees, one in each of the front, back, left, and rightdirections. As will be appreciated, off-road vehicles encounter extremeamounts of vibration and shock when driving on rugged terrain, whichwould cause conventional stereo-vision cameras to misalign quickly.Examples of off-road vehicles include, e.g., military vehicles, jeeps,all-terrain vehicles, dump trucks, mining vehicles, and any type ofvehicle that may be driven on unpaved surfaces such as gravel, rockyterrain, cobblestones, excavation sites, etc. High-resolution 3Dinformation about a surrounding environment of an off-road vehicle maybe critical for navigating uneven surfaces that may be full of rocks,boulders, tree branches, etc., and therefore a reliable system forobtaining such information is of high importance.

FIG. 11 shows two stereo vision systems mounted on a combine harvester1100, according to some embodiments of the present technology. Aforward-facing stereo vision system may include a forward camera 1102and a forward camera 1104, which may be used, e.g., to obtaininformation for measuring an amount of crop ahead to be encountered theharvester 1100, to enable an informed control a throttle of theharvester 1110 and therefore maximize a feed rate. The forward cameras1102, 1104 may, e.g., be mounted on a headliner 1116 of the harvester1100. A spout-mounted stereo vision system may include a spout camera1108 and a spout camera 1110, which may be used, e.g., to control aposition of a spout 1112 and obtain information for measuring an amountof material in a collection container. Both the spout 1112 and theheadliner 1116 of the harvester 1100 may experience extreme amounts ofvibration and shock when the harvester 1100 moves over farmland, plants,and vegetation. Furthermore, as will be appreciated, the headliner 1116and the spout 1112 may not be structures that are “optically rigid” suchthat relative movement of the front cameras 1102, 1104 and relativemovement of the spout cameras 1108, 1110 may be avoided. That is, it maynot be possible to maintain a constant alignment of the pairs of camerasover an agricultural season much less an entire lifetime of theharvester 1100. Camera-alignment issues relevant to the harvester 1100may also be issues in other farm equipment (e.g., tractors, hay balers,earth-mover vehicles, and the like).

FIGS. 12A, 12B, and 12C show a left elevational view, a frontelevational view, and a plan view of cameras 1202, 1204 of aforward-looking stereo vision system mounted on an automated forklift1200, according to some embodiments of the present technology. In someembodiments, the cameras 1202, 1204 may provide a wide field of view of135 degrees, for example, and may enable collision avoidance and objectlocalization, which are desirable features when moving large objectsthat may obstruct an operator's view. Wide baselines of over 1 m inwidth are possible on typical forklift platforms. Forklift systemstypically require a depth range of about 30 m, which is well beyond thecapabilities of conventional stereo-vision solutions. Forklifts also maybe subject to extremely high shocks during operation, which is highlydesirable if not essential for any stereo vision system used withforklifts to have shock-resistant cameras and as well as shock-resistantelectronics (e.g., sensors) that can maintain their calibration. Shockresistance is desirable not only for forklifts but also for automatedmobile vehicles used in, e.g., warehouses and factories (e.g., smartautomatic guided vehicles).

As can be appreciated from the discussions above regarding FIG. 3Athrough FIG. 12, stereo vision systems that may be mounted on anon-rigid platform and that may be calibrated automatically to adjustslow alignment perturbations and/or fast alignment perturbations may bedesirable for many different situations.

FIG. 13 shows a block diagram of a stereo vision system, according tosome embodiments of the present technology. A processing component 1310may be included as part of each of an electronic control unit (e.g., theelectronic control units 212, 214, 216). A first camera 1300 (alsoreferred to as “Camera 1”) and a second camera 1302 (also referred to as“Camera 2”) of the stereo vision system may transmit signals or data ofraw images 1304 and 1306 to the processing component 1310. In someembodiments, the cameras 1300, 1302 may be any of: monochrome CMOScameras, color CMOS cameras, near-infrared cameras, short-wave infraredcameras, mid-wave infrared cameras, long-wave infrared cameras, and mayprovide the raw images 1304 and 1306 to the processing component 1310 inreal time or nearly real time. In some embodiments, the cameras 1300,1302 may have memory and/or may be associated with a data storagedevice, and the cameras 1300, 1302 may to provide a sequence of images(e.g., a sequence of the raw images 1304 and 1306) replayed from thememory and/or the data storage device to the processing component 1310.In some embodiments, the images may be two dimensional (e.g., height andwidth). In some embodiments, the images may be three dimensional (e.g.,height and width and color). The processing component 1310 may beconfigured to be commanded by a main system controller 1316 through acommand and control line 1312. As will be appreciated, the command andcontrol line 1312 may be a wired communication mechanism (e.g., a databus, a communication line) or may be a wireless communication mechanismusing communication techniques known in the art. In some embodiments,the main system controller 1318 may be comprised of a computerconfigured to orchestrate high-level functions (e.g., automaticemergency braking for automobiles), and to communicate with varioussub-systems (e.g., a braking system, a sensor system, and the like). Insome embodiments, common communication protocols may be used forcommunication via the command and control line 1312 (e.g., Ethernet, CAN(Controller Area Network), I2C (Inter-Integrated Circuit), etc.). Insome embodiments, the processing component 1310 may report on acalibration status (e.g., system health status, information qualitystatus, etc.) to the main system controller 1318 through the command andcontrol line 1312. In some embodiments, the command and control line1312 may be used to configure autocalibration and stereo correspondencesettings of the processing component 1310. The command and control line1312 may also be used to start, stop, and record both input and outputdata streams within the processing component 1310. A depth maps 1308 anda corresponding confidence map 1314 may be output from the processingcomponent 1310 for each pair of the raw images 1304 and 1306. In someembodiments, the depth map 1308 may be a two-dimensional matrix with thesame width and height as the raw image 1304, but whose value indicates arange (e.g., a distance) to a target (e.g., an object in the field ofview). The depth map 1308 may be reported or output as an RGB image,where each 24-bit color value may be encoded to be a range to thetarget. For example, if values of the depth map 1308 has units of mm(millimeters), distances from 0 to 16,777.215 m (meters) may berepresented with 24 bits. In some embodiments, the confidence map 1314may be a two-dimensional matrix with the same height and width as theraw image 1304, and each element of the confidence map 1314 mayrepresent a confidence value or confidence level for a depth estimate ofeach pixel of the raw image 1304. In some embodiments, a confidencevalue may be an 8-bit unsigned value from 0 to 255, where relativelyhigher values may indicate a higher level of confidence in acorresponding depth estimate. In some embodiments, using an 8-bitrepresentation for the confidence map 1314 may enable the confidence map1314 to be displayed conveniently as a grayscale image, and also maypermit data for the confidence map 1314 to be transmitted as amonochrome video data stream. Information provided by the confidence map1314 may be important for sensor fusion processing pipelines, whichfunction to combine estimates from various sensors (e.g., radar, lidar,sonar, etc.) to provide high-reliability estimates of a surroundingenvironment.

An illuminator 1316 may be used to project light for nighttimeoperation, according to some embodiments of the present technology. Theilluminator 1316 may be comprised of one or more illumination elements(e.g., lamps, LEDs, etc.). In some embodiments, the stereo vision systemmay be provided on a car, and the illuminator 1316 may be comprised ofthe car's headlights, which may emit visible or infrared radiation. Insome embodiments, the illuminator 1316 may be comprised of externallymounted lamps configured to emit visible or infrared radiation. In someembodiments, the illuminator 1316 may be configured to emit visible orinfrared radiation in synchronization with an exposure interval of afirst camera sensor and an exposure interval of a second camera sensor.With a modulated waveform, a peak optical power may be higher for thesame average optical power, and therefore the number of photonsregistered by the camera sensors may increase without blinding oncomingdrivers or pedestrians, who may be sensitive to the average opticalpower. In some embodiments, the illuminator 1316 may be comprised of avertical cavity surface emitting laser (VCSEL) array, which may beconfigured to project a pseudo-random dot pattern. Use of a randomizeddot pattern may add texture to a scene being illuminated, which mayimprove the ability of a stereo correspondence engine 1420 to generateaccurate depth maps.

In some embodiments of the present technology, the stereo vision systemof FIG. 13 may be modified to be a three-camera system. A first cameraand a second camera may form a stereo vision system for short-rangesensing, and the first camera and a third camera may form a stereovision system for long-range sensing. A baseline distance between thefirst and second cameras may be shorter than a baseline distance betweenthe first and third cameras. The stereo vision system comprised of thefirst and second cameras may have a range with a lower minimum distancebut may have a higher range resolution compared to the stereo visionsystem comprised of the first and third cameras.

In some embodiments of the present technology, the stereo vision systemof FIG. 13 may be modified to be a four-camera system comprised of twostereo vision systems: a short-range system with a first camera and asecond camera, and a long-range system with a third camera and a fourthcamera. In comparison with the three-camera system, the additional(fourth) camera may provide an additional degree of operational freedomand may allow the short-range system to have a different (e.g., wider)field of view than the long-range system. In some embodiments, the firstand second cameras may have a shorter baseline and a wider field of viewthan the third and fourth cameras.

In some embodiments of the present technology, the stereo vision systemof FIG. 13 may be modified to include a time-of-flight camera, which maybe configured to measure objects that are closer to the cameras 1300,1302 than a minimum distance of the stereo vision system.

FIG. 14 shows a block diagram of the processing component 1310,according to some embodiments of the present technology. As discussedabove, the processing component 1310 may convert the raw images 1304 and1306 into depth maps 1308 and confidence maps 1314. Raw images 1304 fromthe first camera may be rectified by a rectification engine 1416. Rawimages 1306 from the second camera may be rectified by a rectificationengine 1418. As will be appreciated, in computer-vision technology,rectification is a transformation process used to warp images so thatimages from Camera 1 and Camera 2 are aligned row-wise. That is,features in a row of pixels in a rectified image 1406 produced from theraw image 1304 from Camera 1 are aligned to a same row of pixels asmatching features in a rectified image 1408 from the raw image 1306 ofCamera 2. Such row alignment of pixels may allow the stereocorrespondence engine 1420 to operate faster, because it enables asearch for corresponding features to be a one-dimensional search alongrows rather than a two-dimensional search along rows and columns. Insome embodiments, the rectification engines 1416 and 1418 may utilizecamera parameters 1410 and 1412, respectively, to determine a mapping orwarping operation. FIG. 15A shows a flow diagram of a rectificationalgorithm that may be utilized by the rectification engines 1416, 1418,in some embodiments of the present technology. FIG. 15B shows acoordinate system and geometric details of a rectification procedureperformed by the algorithm, according to some embodiments of the presenttechnology.

According to some embodiments of the present technology, the cameraparameters 1410, 1412 may be comprised of six (6) extrinsic cameraparameters, which may describe a relative position and orientation ofthe two cameras 1300, 1302, and eighteen (18) intrinsic cameraparameters, which may characterize a transformation from a cameracoordinate system to a pixel coordinate system of each camera 1300,1302. The six extrinsic parameters may be comprised of: a relative roll,a relative pitch, a relative yaw, a relative translation x, a relativetranslation y, and a relative translation z. The eighteen intrinsiccamera parameters for both cameras 1300, 1302 may be comprised of, foreach camera 1300, 1302: focal length (Fx, Fy); principal point (Cx, Cy);radial lens distortion coefficients K1, K2, and K3; and tangential lensdistortion coefficients P1 and P2. These camera parameters are wellknown and are described by OpenCV (Open Source Computer Vision Library),which is an open-source library that includes hundreds ofcomputer-vision algorithms. When a wide field-of-view lens and/or afisheye lens is used, one or more additional higher-order lensdistortion parameters may need to be used. In some embodiments, thecamera parameters 1410, 1412 may not be constant but may change as afunction of time, to track actual changes in the cameras 1300, 1302. Insome embodiments, the camera parameters 1410, 1412 may be updated forevery frame.

According to some embodiments of the present technology, anautocalibration engine 1414 may be configured to implement a method thatmaintains stereo camera calibration of the cameras 1300, 1302. It someembodiments, the autocalibration engine 1414 may utilize as inputs theraw images 1304, 1306 of a current frame and the rectified images 1406,1408 of a previous frame. The autocalibration engine 1414 may output thecamera parameters 1410 for the first camera 1300 (Camera 1) and thecamera parameters 1412 for the second camera 1302 (Camera 2), and mayoutput translation parameters 1426 for the first camera 1300 andtranslation parameters 1428 for the second camera 1302.

According to some embodiments of the present technology, translationengines 1422, 1424 may perform a transformation operation to shift therectified images 1406, 1408 to compensate for very fast perturbationsand to stabilize the rectified images 1406, 1408 with respect to eachother. The translation operation may be performed by pixel row and/orpixel column, which may be accomplished quickly using an affinetransformation procedure. The translation parameters 1426, 1428 may berow-based and column-based translation offsets. In some embodiments, toreduce computation time, one of the translation engines 1422, 1424 maybe dropped (e.g., unused or idle or absent), while the other receivesthe translation offsets (e.g., as a sum). Images 1430, 1432 output bythe translation engines 1422, 1424 may be rectified and stabilizedimages 1430, 1432, which may be fed into the stereo correspondenceengine 1420. The stereo correspondence engine 1420 may determine adisparity between matching pixels in the images 1430, 1432 (e.g., theimages 1430, 1432 may be left and right images), may compute a disparitymap, may compute and output the depth map 1308 from the disparity map,and may compute and output the confidence map 1314.

FIG. 16 shows a block diagram of the autocalibration engine 1414,according to some embodiments of the present technology. Theautocalibration engine 1414 may be comprised of five calibrationengines: a fast calibration engine 1600, a medium calibration engine1602, a slow calibration engine 1604, an absolute range calibrationengine 1606, and a stereo image stabilization engine 1608. In someembodiments, the fast 1600, medium 1602, and slow 1604 calibrationengines may compensate for camera parameter perturbations that may occurover slow to fast timescales, and the methods used by these calibrationengines 1600, 1602, 1604 may be based on optimizing a cost function. Insome embodiments, the absolute range calibration engine 1606 maycalibrate an absolute range to objects by adjusting an extrinsic cameraparameter: relative yaw. In some embodiments, calibration of theabsolute range calibration may be one of the most important and hardestparameters to calibrate well because epipolar geometry constraints maybe insensitive to changes in yaw. In some embodiments, the stereo imagestabilization engine 1608 may be used to track and compensate forframe-to-frame variations in pitch and yaw. A time-series history ofcamera parameters may be stored in a storage device 1622. Current cameraparameters 1620 may be used as a starting point for the calibrationengines 1600, 1602, 1604, 1606, 1608. In some embodiments, outputs 1610,1612, 1614, 1616, 1626 of the calibration engines 1600, 1602, 1604,1606, 1608 may be combined by a calibration manager 1624, which usesthis information to decide a best estimate of the camera parameters1410, 1412 for the first camera and second cameras 1300, 1302. Thecommand and control line 1312 may be used to initialize values for thecamera parameters, which may come from a manual factory calibration,manufacturing specifications, or a random guess. As discussed below, theabsolute range calibration engine 1606 have different embodiments. Insome embodiments, any one or any combination of a fast calibrationprocedure performed by the fast calibration engine 1600, a mediumcalibration procedure performed by the medium calibration engine 1602,and a slow calibration procedure performed by the slow calibrationengine 1604 may include an intrinsic-parameter calibration procedure forcalibrating one or more intrinsic camera parameter(s). For example, foreach camera, any one or any combination of: a focal length, a principalpoint, at least one radial lens distortion coefficient, and at least onetangential lens distortion coefficient may be calibrated by theintrinsic-parameter calibration procedure. In some embodiments, theintrinsic-parameter calibration procedure may be performed by the fastcalibration engine 1600 and/or the medium calibration engine 1602 ifthere is a priori knowledge about the first camera and second cameras1300, 1302, for example, if it is known that a lens mount itself isflexible, which may be a rare case. In some embodiments, the intrinsiccamera parameters may not need to be calibrated on the fast timescaleand/or the medium timescale for a number of reasons: (1) small changesin the piercing point horizontal location (Cx) and focal length (Fx andFy) may be corrected to first order by changes to yaw, (2) small changesin the piercing point vertical location (Cy) may be corrected to firstorder by changes to pitch, and (3) the lens distortion coefficients maynot typically change since the lens shape may be generally immutable.

FIG. 17A shows a block diagram of the fast 1600, medium 1602, and slow1604 calibration engines, according to some embodiments of the presenttechnology. The calibration engines 1600, 1602, 1604 may have the sameblock-diagram structure but may utilize different optimization methods1700, 1722, 1724, respectively. In some embodiments, a basic concept ofthe methods 1700, 1722, 1724 is that, for each of the methods 1700,1722, 1724, optimal or correct camera parameters may be associated witha minimized cost function 1704 appropriate for the corresponding method.For example, for a nominal case of twenty four (24) camera parameters(six extrinsic parameters and eighteen intrinsic parameters), abrute-force search of parameters may require evaluating a cost function100²⁴ times, assuming 100 values per parameter, which take anunreasonably long time to compute using conventional computers (it wouldtake longer than the age of the universe using conventional computers).As will be appreciated, to be practical for a real-world setting, anoptimization operation should be completed in real time and implementedon a time scale that matches a time scale of a source or cause of aperturbation, or faster. Therefore, in some embodiments, it is highlydesirable for the camera parameters to be searched efficiently byseparating the search into different time scales.

According to some embodiments of the present technology, one or more ofthe optimization methods 1700, 1722, 1724 may repeatedly evaluate thecost function 1704 for a subset of camera parameters over a boundedsearch range. In some embodiments, to speed up evaluation of the costfunction 1704, one or more of the optimization methods 1700, 1722, 1724may dynamically tune a pyramid level 1703 of the raw images 1304, 1306from the first and second cameras 1300, 1302. The term “pyramid” is acommon computer-vision term that is widely understood to refers todownsampling of images to smaller images. Because an amount of time tocompute the cost function (i.e., determine a minimum cost function) maybe proportional to a size of an image being processed, by subsampling tohalf resolution (or pyramid level 1), the time to compute the costfunction may be reduced by a factor of four (4). For coarse parametersearches, high pyramid levels may be used to speed up searching. Forfine parameter searches, low pyramid levels may be used for enhancedaccuracy. In some embodiments, the respective outputs 1610, 1612, 1614of the optimization methods 1700, 1722, 1724 may be the cameraparameters that minimize the cost function 1704 of the correspondingoptimization method. As will be appreciated, minimizing a cost functionmay also be considered maximizing a figure of merit, as they may beconsidered equivalent approaches.

According to some embodiments of the present technology, an indicator ofcalibration health may be reported to the main system controller 1316through the command and control line 1312. In some embodiments, thecalibration health may be taken to be a negative of the (minimized) costfunction (or the maximized figure of merit) corresponding to optimizedcamera parameters.

According to some embodiments of the present technology, the costfunction 1704 may be comprised of a pyramid down block 1701 configuredto reduces a size of the raw images 1304, 1306, a rectification block1706 configured to warp and row-align reduced images according to testcamera parameters 1702, a stereo block matching block 1710, and a block1714 configured to compute a negative of a number of valid pixels. Insome embodiments, the stereo block matching block 1710 may use a stereoblock matching algorithm from OpenCV with following parameters that mayinclude: a pixel-window size of 9×9, a normalized response prefilter, aprefilter cap of 63, a prefilter size of 11, a speckle window size of100, a speckle range of 64, a texture threshold of 10, a uniquenessratio of 10, and a disparity search range of 0 to 255. Such a blockmatching algorithm may be chosen for its fast execution across manydifferent types of processors. A texture threshold and a uniquenessratio may be set relatively high (e.g., a value of 10-50) to minimize anumber of wrong distance estimates and to replace the correspondingpixels with invalid pixels. In some embodiments, each of theoptimization methods 1700, 1722, 1724 may find optimal camera parametersthat minimize the number of invalid pixels after stereo block matchingis performed.

FIG. 17B shows a flow diagram of the fast optimization method 1700 ofthe fast calibration engine 1600, according to some embodiments of thepresent technology. The fast optimization method 1700 may tune pitch androll extrinsic camera parameters, because these parameters may beresponsible for most of short-timescale distortions. The fastoptimization method 1700 may be used for the most frequent updates amongthe optimization methods 1700, 1722, 1724. The fast optimization method1700 may be considered fast because it searches a relatively smallnumber of camera parameters (two) over a small range.

At step 1726 of the fast optimization method 1700, a quick search isperformed for an initial guess of the camera parameters 1620 by checkingcosts in a chosen grid points. In some embodiments, the grid points maytypically have nine values in pitch from −0.4 degrees to 0.4 degrees andthree values in roll from −0.2 degrees to 0.2 degrees about the initialguess of the camera parameters 1620, for a total of 27 grid points. Forevaluating the cost function 1704, a pyramid level of 1 may be used tospeed up the evaluation.

At step 1728 of fast optimization method 1700, a pitch and a rollcorresponding to a lowest cost (a minimum of the cost function 1704) maybe selected among the grid points.

At step 1730 of fast optimization method 1700, a COBYLA (ConstrainedOptimization BY Linear Approximations) algorithm for derivative-freeoptimization with nonlinear inequality constraints may be applied withthe pitch and the roll found in step 1728, other camera parameters ofthe initial guess of the camera parameters 1620, and bounds of 0.1degrees in both the pitch and the roll. The cost function 1704 used instep 1730 may have a pyramid level of 0. The camera parameters 1620 maybe updated with the pitch and the roll corresponding to a lowest costfound by COBYLA, and updated camera parameters 1610 are returned for thefast optimization method 1700.

According to some embodiments of the present technology, the COBYLAalgorithm may construct successive linear approximations of an objectivefunction and constraints with a simplex of points and may optimize theseapproximations in a trust region at each step. In some embodiments,because there are only two search dimensions, the relative pitch androll of the pair of stereo cameras, over a small search range of 0.1degrees for pitch and 0.1 degrees for roll, optimization can be computedquickly (e.g., 100 ms on an Intel Core i7 CPU processor) and cancompensate every other frame at 20 FPS, if needed. In some embodiments,a search range can be increased or decreased based on expectedexcursions for a given platform. For example, for suction cup mounts ona car, 0.1 degrees is large enough, whereas for loosely mounted camerason a mountain bike ridden on a trail, 0.2 degrees might be needed.

It should be noted that in some embodiments, the COBYLA search methodcould also be replaced by other search methods, such as the Nelder-Meadsimplex search method, Broyden-Fletcher-Goldfar-Shanno (BFGS) method,Powell's method, Sequential Least Squares Programming (SLSQP) method, orMonte Carlo methods.

FIG. 17C-1 shows a flow diagram of the medium optimization method 1722of the medium calibration engine 1602, according to some embodiments ofthe present technology. The medium optimization method 1722 may updatemore parameters than the fast optimization method 1700 but update slowerthan the fast optimization method 1700. In some embodiments, a mainobjective of the medium optimization method 1722 is to find atranslation vector and then to optimize pitch and roll. FIG. 17C-2 showsa flow diagram of an angle-search process of the medium optimizationmethod of FIG. 17C-1, according to some embodiments of the presenttechnology.

At step 1732 of the medium optimization method 1722 a grid search isperformed over translation vector angles, which are represented by twoangles: ay and az. The angle ay may be an angle between the translationvector and the x-axis measured on the xz plane. The angle az may be anangle between the translation vector and the x-axis measured on the yzplane. For example, the search may test all of the grid of values (ay,az) from −7 degrees to 7 degrees in 1-degree increments around aninitial translation vector of the camera parameters 1620. For eachtranslation vector, step 1732 uses an angle-search process 1736 (seeFIG. 17C-2) to look for a best tuple of pitch and roll angles that yielda lowest cost. In some embodiments of the present technology, theangle-search process 1736 may break the search process into coarse 1738,medium 1740, and fine 1742 angle searches.

The coarse angle search 1738 may apply nine (9) COBYLA searches,centered at (pitch, yaw, and roll) of (0,0,0), (−1,−1,−1), (−1,−1,1),(−1,1,−1), (1,−1,−1), (1,1,−1), (1,−1,1), (−1,1,1), and (1,1,1) degreesfrom the initial pitch, yaw, and roll specified in the camera parameters1620.

The COBYLA search may use pyramid level 3 with the cost function 1704for a fast evaluation, and the search bounds on all angles may be [−1,1]degrees. The pitch, yaw, and roll corresponding to the lowest cost maybe used as the starting point for the medium angle search 1740.

The medium angle search 1740 may apply a COBYLA search starting from thepitch, yaw and roll returned from the coarse angle search 1738. ThisCOBYLA search may have angular bounds of [−0.2, 0.2] degrees for pitch,yaw, and roll, and may use pyramid level 3 for the cost function 1704.The pitch, yaw, and roll corresponding to the lowest cost may be used asthe starting point for the fine angle search 1742.

The fine angle search 1742 may apply a COBYLA search starting from thepitch, yaw, and roll returned from the medium angle search 1740. ThisCOBYLA search may have angular bounds of [−0.1, 0.1] degrees for pitch,yaw, and roll, and may use pyramid level 0 for the cost function 1704.The pitch, yaw, and roll corresponding to the lowest cost may bereturned to step 1732.

The best estimate for ay, az, pitch, yaw, and roll from step 1732 may beused as a starting point for a fine translation vector and angle search1734, which may apply a COBYLA optimization for (ay, az) followed by aCOBYLA optimization for (pitch, yaw, roll). Both COBYLA searches may usepyramid level 0 for the cost function 1704 to maximize sensitivity tothe optimization parameters. The two COBYLA searches may be repeateduntil little improvement is seen in the cost function. The initialcamera parameters 1620 may be updated with the optimal values for ay,az, pitch, yaw, and roll, and then the updated camera parameters 1612may be returned from the medium optimization method 1722.

FIGS. 17D-1 through 17D-4 (collectively “FIG. 17D”) show flow diagramsof a slow optimization method 1724, according to some embodiments of thepresent technology. In some embodiments, the slow optimization method1724 of FIG. 17D may be performed by the slow calibration engine 1604once at factory calibration and then when deemed necessary by the mainsystem controller 1316 (e.g., after an accelerometer or other sensorreports a large shock event). The slow optimization method 1724 mayoptimizes the extrinsic camera parameters and the intrinsic cameraparameters. The step 1744, a global grid search may be performed overthe pitch, yaw, and roll angles with a sampling interval (e.g., 0.2degrees, 0.4 degrees, 0.7 degrees, etc.) over [−10, 10] degrees. Theremaining steps 1746 through 1796 FIG. 17D may apply local searchmethods, such as COBYLA, to a previously best estimate of the cameraparameters 1602. As noted above, the intrinsic camera parameters may bethe focal lengths (FX, FY), the piercing points (CX, CY), and the lensdistortion coefficients (K1, K2, K3, P1, P2). The extrinsic cameraparameters may be the pitch, the yaw, the roll, and the translationvector (TX, TY, TZ). In some embodiments, optimization of the pitch, theyaw, and the roll may be followed by optimization of the translationvector from step 1746 to step 1748, from step 1768 to step 1770, fromstep 1772 to step 1774, from step 1784 to step 1786, and from step 1788to step 1790, because complementary parameters (e.g., pitch and TX; rolland TY; yaw and TZ) may warp the images (e.g., the raw images 1304,1306) similarly.

FIG. 18 shows a flow diagram for the stereo image stabilization engine1608 of the autocalibration engine 1414, according to some embodimentsof the present technology. The stereo image stabilization engine 1608may determine an amount to horizontally and vertically shift imagesobtained from Camera 1 and Camera 2 so that a relative displacementbetween these cameras is minimized. The stereo image stabilizationengine 1608 may compensate for fastest perturbations (e.g., engine noiseand road noise) and may update every frame. Unlike conventionalmonocular image stabilization, which may stabilize an image with respectto the Earth, the stereo image stabilization engine 1608 may stabilizetwo images with respect to each other. In some embodiments, the stereoimage stabilization engine 1608 receives as input the rectified images1406 produced from the raw images 1304 from Camera 1 and the rectifiedimages 1408 produced from the raw images 1306 of Camera 2. The stereoimage stabilization engine 1608 may output translation parameters 1426for Camera 1 and translation parameters 1428 for Camera 2. At 1800 aforward tracking process may compute an optical flow for a sparsefeature set using an iterative Lucas-Kanade method with pyramids for therectified images 1406, 1408, which may be left and right rectifiedimages, from a previous frame to a current frame. At 1802 a backwardtracking process may compute an optical flow in a reverse direction,from points in the current frame to the previous frame. At 1804 aprocess may determine good tracks from back tracks (e.g. a good track iswhen the backward tracking process at 1802 reprojects a point within 1pixel of a previous point of the track). At 1806 a process may drop thebad tracks. At 1808 a process may add remaining good points to thetracks. At 1810 a process may update an estimate of the translationparameters. In some embodiments, the translation parameters 1426 forCamera 1 and the translation parameters 1428 for Camera 2 may be amountsto shift rows and columns of images from Camera 1 and Camera 2 relativeto a previous frame. In some embodiments, the translation parameters maybe equal to an average relative translation between images from Camera 1and Camera 2 from frame-to-frame with outliers removed from the average.

After several frames, one or more tracks may be dropped becausekeypoints may move off screen (e.g., out of the field of view) or may beobstructed. At 1812 a process may find new keypoints to create newtracks. For example, every n frames, where n may typically be 5, thestereo image stabilization engine 1608 may search for new keypoints inthe rectified images 1406, 1408. A keypoint detector (not shown) maylook for good points to track using, for example, the Shi-Tomasi cornerdetector. At 1814 a process may translate the rectified images 1406,1408 using the translation parameters 1426, 1428 to remove any relativemotion between Camera 1 and Camera 2. At 1816 a process may compute adisparity map from the rectified images 1406, 1408 using a blockmatching algorithm. At 1818 a process may ignore currently trackedkeypoints for the rectified image 1406 of Camera 1, and at 1820 aprocess may find good features to track in the rectified image 1406 ofCamera 1. At 1822, for each keypoint found for the rectified image 1406of Camera 1, a process may find a matching keypoint in the rectifiedimage 1408 of Camera 2 using processes at 1824, 1826, 1828, 1830. At1824 a process may be performed for creating a Hanning window for aphasecorrelation function. In OpenCV, phasecorrelation refers to amethod to check the similarity of two images with equal size.Phasecorrelation is a fast implementation of template matching. At 1826a process may be performed to find matching keypoints between Camera 1and Camera 2. Since images 1406 and 1408 are rectified, matchingkeypoints lie along the same row. Adding the disparity values 1816 tothe Camera 1 keypoint column values yield the expected location for theCamera 2 keypoint column values. At 1828 a process may be performed toverify the matches of 1826 by computing the phasecorrelation functionfor the matched keypoints. The phase correlation function is computedover image patches (typically 40×40 pixels) centered on the pairs ofmatching keypoints between Camera 1 and Camera 2. The phasecorrelationfunction returns the corrected coordinate shift and the response(typically, a value between 0 and 1 that indicates the similarity of thetwo image patches). At 1830 a process is performed to record thematching keypoints, correct the keypoint coordinates according to thecorrected coordinate shift from 1828, and update the track informationif the phasecorrelation response is greater than a threshold value(typically, a threshold value of 0.3).

In some embodiments of the present technology, the absolute rangecalibration engine 1606 may have three different implementations(referred to as 1606 a, 1606 b, and 1606 c) any or all of which may beincorporated in the autocalibration engine 1414. The threeimplementations 1606 a, 1606 b, and 1606 c may use different absolutedistance references: the implementation 1606 a may use a non-negativedisparity constraint (see FIGS. 19A, 19B), the implementation 1606 b mayuse a known object such as a stop sign (see FIGS. 20A, 20B), and theimplementation 1606 c may use vehicle odometry (see FIGS. 21A, 21B). Insome embodiments, a relative yaw between the cameras 1300, 1302 may havea large influence on calibration of an absolute range of the stereovision system. The absolute range calibration engine 1606 may be used tocalibrate the yaw.

FIG. 19A shows a flow diagram of the implementation 1606 a, in which theabsolute range calibration engine 1606 has a non-negative disparityconstraint, according to some embodiments of the present technology. Theimplementation 1606 a may find the yaw where principal rays of the firstand second cameras 1300, 1302 do not cross. When the principal rayscross in front of the cameras, a corresponding disparity value may benegative at ranges farther than the crossing point. By tuning the yawuntil the rays do not cross, the cameras 1300, 1302 may become parallel,a number of negative values in the disparity map 1712 may decreases(sometimes drastically) to a minimum value, and a correct absolute rangemay be found.

FIG. 19B is a chart 1900 showing how a number of pixels with a negativedisparity value in a disparity map (e.g., the disparity map computedfrom raw images 1304, 1306 after rectification using the initial cameraparameters 1620 but with various values of yaw) can vary as a functionof yaw, according to some embodiments of the present technology. For yawvalues less than approximately 1.8 degrees in the chart 1900, the numberof negative disparity values greatly increases, indicating that theprincipal rays of the cameras 1300, 1302 may start to cross at a longrange and then at increasingly closer range. For yaw values less than1.65 degrees, the number of negative disparity values decreases. Thismay be because a block matcher used in this analysis may only searchdisparity values down to −32 pixels. Any pixel with disparity values of−33 pixels or lower may not be counted. For yaw values greater than 1.85degrees, the number of pixels with negative values is small but notzero. This may be because mismatching pixels sometimes may have negativedisparity values. A desired yaw is in a transition region where thechart 1900 transitions from a region where the number of pixels is smalland generally constant (yaw>1.85 degrees in the chart 1900) and includesa region where the number of pixels is rapidly increasing (e.g., aregion indicated by an upward pointing arrow 1912). The chart 1900 alsoshows a region in which disparity values of pixels may be more negativethan a minimum disparity limit (e.g., a region indicated by a downwardpointing arrow 1914). Returning to the flow diagram of theimplementation 1606 a shown in FIG. 19A, at 1920 a coarse search isperformed from −0.5 degrees (see 1902) to 1.0 degrees (see 1906) aroundan initial guess 1904 for the yaw for which the disparity map iscomputed, and the number of negative disparity values is tallied. Thesearch may be performed in 0.1-degree steps. A result of the coarsesearch may be an array of sixteen (16) values. The chart 1900 is anexample of a plot of sixteen datapoints for a coarse curve. A derivativeof the coarse curve with respect to yaw may be computed. The derivativeof the coarse curve may be sensitive to changes and therefore may beuseful for finding transitions in the coarse curve. At 1922 a quality ofthe coarse search is checked by making sure that the transition region(e.g., the region from 1.65 deg to 1.8 deg in the chart 1900) is greaterthan five times a tail noise 1908. The tail noise 1908 may be defined tobe a standard deviation of the last 5 points of the derivative of thecoarse curve. In other words, the transition region should have asignificant change in the number of negative disparity values comparedto a region where the number of negative disparities should be zero.Next, at 1924 a fine search range is determined by searching around theyaw corresponding to the most negative slope (e.g., a yaw of −1.7degrees in the chart 1900). Starting from the largest coarse yaw valueand searching toward lower coarse yaw values, an upper bound isdetermined to be a value where the derivative of the coarse curve isgreater than −3 times the tail noise. Starting from the yawcorresponding to the most negative slope and increasing to the largestyaw, the lower bound is determined to be a value where the derivative ofthe coarse curve is less than −3 times the tail noise. Next, at 1926 afine search grid is created from the lower to upper bounds, and a finesearch is performed. This grid may typically have twenty-one (21)points. At 1928 a yaw value corresponding to a minimum number ofnegative disparity pixels is determined. In some embodiments, atransition point in the curve may be estimated in this manner.

FIG. 20A shows a flow diagram of the implementation 1606 b, in which theabsolute range calibration engine 1606 uses an object with knowndimensions, according to some embodiments of the present technology.FIG. 20B illustrates an example of imaging optics for imaging an object2006 with known dimensions, according to some embodiments of the presenttechnology. In some embodiments, because the dimensions of the object2006 are known and because focal lengths of camera lenses of the cameras1300, 1302 are known, the range or distance to the object (target) 2006may be determined by an equation 2020. More specifically, according toan equation 2020, a ratio of a width W of the object (see 2012) to arange R to the object (see 2014) is equal to a ratio of a focal length Fof a camera lens 2008 (see 2016) to a width H of an image (see 2018)sensed by a camera sensor 2010 (e.g., a CMOS sensor). In someembodiments, a known object can be recognized in an automated fashionusing known detection technology (e.g., a traffic sign recognitionsystem, license plate detector, an object detector, etc.). In someembodiments, a known object may be specified manually by a manual inputof a bounding box and a distance to the known object. An example of anobject detector for detecting stop signs is shown in FIG. 24.

Returning to the flow diagram of the implementation 1606 b shown in FIG.20A, at 2000 the absolute range calibration engine 1600 finds the yawfor which the mean distance to the object equals the known distance. Insome embodiments of the present technology, the mean distance may becomputed over a region of interest of the object. At 2002 a goodstarting interval for a root finding algorithm is found by firstinitializing an interval to ±0.1 degrees of the yaw of the initialcamera parameters 1620, and then modifying left and right ends by 0.1degrees until a mean depth over the region of interest is positive atthe left end of the interval and negative at the right end of theinterval. At 2004 the root finding algorithm is run. In someembodiments, the root finding algorithm may be Brent's method, in whichwith the search interval found at 2002 is used to determine a bestestimate of yaw. Subsequently, the initial camera parameters 1620 arereturned with an updated yaw value 1616.

FIG. 21A shows a flow diagram of the implementation 1606 c, in which theabsolute range calibration engine 1606 utilizes vehicle odometry 1628,according to some embodiments of the present technology. FIG. 21B is achart 2108 showing how disparity can vary as a function of range,according to some embodiments of the present technology. Odometry (e.g.,speed, distance, heading, etc.) may be information already availablefrom in-car (“on-board”) sensors, and may be used to help calculate theyaw of the stereo vision system. In some embodiments, odometry may beavailable from on-board radar and/or lidar systems. In theimplementation 1606 c, at 2100, for each track 1626 from the stereoimage stabilization engine 1608, a change in range, ΔR_(ODO), iscomputer from odometry data from a time t1 to time a t2. A correspondingchange in range, ΔR, is measured by the stereo vision system and mightnot match ΔR_(ODO) because of miscalibration. At 2102 a disparityoffset, d_(offset), is computed using a function expressed by: d=fB/R,where f is the focal length of the lens and B is the baseline widthbetween the first and second cameras 1300, 1302, so that a correctedrange ΔR′ equals ΔR_(ODO). Tracks of stationary objects may give thesame or similar values for d_(offset) but moving objects may not. At2104 outlier values for d_(offset) are removed and remaining values areaveraged. At 2104 yaw is calibrated to be equal to the initial yaw plusthe average disparity offset divided by the focal length of the cameras.Subsequently, the initial camera parameters 1620 are returned with anupdated yaw value 1616.

FIG. 22 shows a flow diagram of a procedure performed by the calibrationmanager 1624, according to some embodiments of the present technology.At 2200 camera parameters are initialized the command and controlinterface 1312 configures the calibration engines 1600, 1602, 1604,1606, 1608. The initial camera parameters may be extrinsic and intrinsiccamera parameters obtained from blueprints, CAD drawings, manufacturer'sdatasheets, etc., or may be obtained from manual calibration procedures.At 2202 the calibration manager 1624 waits for outputs from thecalibration engines 1600, 1602, 1604, 1606, 1608. Once an output of anestimate of a camera parameter is received, at 2204 the estimate is usedin a tracking filter (e.g., a Kalman filter) that uses a series ofmeasurements over time and produces a new estimate of the cameraparameter, which may be more accurate than an estimate on a singlemeasurement alone. This may be done for each of a plurality of cameraparameters. At 2206, new estimates for the camera parameters may besaved in the storage device 1622 and the updated camera parameters 1410,1412 for Camera 1 and Camera 2 may be output from the calibrationmanager 1624.

FIG. 23 shows a time series input diagram of the calibration manager1624, according to some embodiments of the present technology. The timeaxis is shown for different frame numbers, which may correspond todifferent frame rates (e.g., 10, 30, or 60 FPS). Upward arrows indicatethe initialization or start of a calibration method 1600, 1602, 1604,1606, 1608 by the calibration manager 1624, and the downward arrowsindicate the completion of the calibration method, where optimalparameters are returned to the calibration manager 1624. The executiontimes for calibration methods in FIG. 23 is an example of oneembodiment.

FIG. 24 shows a flow diagram of a procedure that may be performed by astop sign detector 2400 that may be used for absolute range calibration,according to some embodiments of the present technology. Input to thedetector 2400 may be a color image 2402. For example, the detector 2400may look for an arrangement of red connected components in the letters“STOP.” These components may include a red region inside an oval (e.g.,the letter “0” 2406), a red region inside a particular shape (e.g., theletter “P” 2408), and a red octagonal region 2404. The relativelocations and sizes of the connected components may give a uniquesignature, which may be detected quickly with few computationalresources.

FIG. 25 shows a flow diagram of a procedure performed by the stereocorrespondence engine 1420, according to some embodiments of the presenttechnology. Input to the stereo correspondence engine 1420 may be therectified and stabilized images 1430, 1432 from Camera 1 and Camera 2.The stereo correspondence engine 1420 may output depth maps 1308 andconfidence maps 1314. In some embodiments, a matching window 2500 in therectified and stabilized image 1430 may be compared to a correspondingrow 2502 in the rectified and stabilized image 1432. For each pixel, aresulting matching value 2506 may be a negative value of a sum ofabsolute differences between the matching window and each offset alongthe corresponding row 2502. The matching value for each offset may becomputed, where a best match 2510 may correspond to a highest matchingvalue and where a second-best match 2512 may correspond to asecond-highest matching value. A uniqueness ratio for a given pixel maybe a ratio determined as the best match 2510 divided by the second-bestmatch 2512. A higher confidence corresponds to a higher uniquenessratio. The depth map 1308 may be derived from the best match 2510. Theconfidence map may be derived from the ratio of best match 2510 tosecond-best match 2512.

According to some embodiments of the present technology, the processingcomponent 1310 may implemented in hardware (e.g., a computer processorprogrammed to perform the procedures and methods described above).According to some embodiments of the present technology, the processingcomponent 1310 may be implemented in software (e.g., computer-executablecode), which may be stored on a non-transitory computer-readable storagemedium or on a plurality of non-transitory computer-readable storagemedia, and which may be accessed and executed by a computer processor.According to some embodiments of the present technology, the processingcomponent 1310 may be implemented in a combination of hardware andsoftware. In some embodiments, aspects of the processing component 1310may be implemented as one or more software modules. For example, one ofmore the engines of the processing component 1310 may be implemented assoftware module(s) stored on a non-transitory computer-readable storagemedium.

A stereo imaging system according to the technology described herein maybe embodied in different configurations. Example configurations includecombinations of configurations (1) through (27), as follows:

-   -   (1) A stereo vision system, comprising: a first camera sensor        configured to sense first reflected energy of a first image and        to generate first sensor signals based on the first reflected        energy; a second camera sensor configured to sense second        reflected energy of a second image and generate second sensor        signals based on the second reflected energy; at least one        processor configured to receive the first sensor signals from        the first camera sensor and the second sensor signals from the        second camera sensor, and to produce three-dimensional (3D) data        from the first and second sensor signals, wherein the at least        one processor is configured to: generate rectified stereo images        from the first and second sensor signals utilizing stereo        calibration parameters, perform a stereo matching on the        rectified images, and perform an automatic system calibration        using data from a plurality of stereo images obtained by the        first and second camera sensors, wherein the automatic system        calibration is based on minimizing a cost function.    -   (2) The stereo vision system of configuration (1), wherein the        automatic system calibration is comprised of a fast calibration        procedure configured to generate first stereo calibration        parameters to perform one or both of:    -   to compensate for high-frequency perturbations based on tracking        image points corresponding to features in the first and second        images, and    -   to calibrate, for each camera, any one or any combination of: a        focal length, a principal point, at least one radial lens        distortion coefficient, and at least one tangential lens        distortion coefficient.    -   (3) The stereo vision system of any of configurations (1) to        (2), wherein the automatic system calibration is comprised of a        medium calibration procedure configured to generate second        stereo calibration parameters to perform one or both of:    -   to compensate for medium-frequency perturbations based on        optimizing a number of valid pixels for dense stereo block        matching as a function of a relative camera pitch and a relative        camera roll, and    -   to calibrate, for each camera, any one or any combination of: a        focal length, a principal point, at least one radial lens        distortion coefficient, and at least one tangential lens        distortion coefficient.    -   (4) The stereo vision system of any of configurations (1) to        (3), wherein the automatic system calibration is comprised of a        slow calibration procedure configured to generate third stereo        calibration parameters to perform one or both of:    -   to compensate for low-frequency perturbations based on        optimizing a number of valid pixels for dense stereo block        matching as a function of any one or any combination of: a        relative camera pitch, a relative camera roll, a relative camera        yaw, a relative camera position, and    -   to calibrate, for each camera, any one or any combination of: a        focal length, a principal point, at least one radial lens        distortion coefficient, and at least one tangential lens        distortion coefficient.    -   (5) The stereo vision system of any of configurations (1) to        (4), wherein the automatic system calibration is comprised of an        absolute range calibration procedure configured to generate a        stereo calibration parameter for a relative camera yaw based on        a non-negative disparity constraint.    -   (6) The stereo vision system of any of configurations (1) to        (5), wherein the automatic system calibration is comprised of an        absolute range calibration procedure configured to generate a        stereo calibration parameter for a relative camera yaw based on        dimensions of an object in a scene of a plurality of stereo        images and a focal length of the first and second camera        sensors.    -   (7) The stereo vision system of configuration (6), wherein the        object is a traffic sign.    -   (8) The stereo vision system of configuration (6), wherein the        object is a license plate.    -   (9) The stereo vision system of any of configurations (1) to        (8), wherein the automatic system calibration is comprised of an        absolute range calibration procedure configured to generate a        stereo calibration parameter for a relative camera yaw based on        vehicle odometry.    -   (10) The stereo vision system of any of configurations (1) to        (9), wherein the automatic system calibration is comprised of a        calibration management procedure configured to manage an        operation schedule of the slow calibration procedure, the medium        calibration procedure, the fast calibration procedure, and an        absolute range calibration procedure.    -   (11) The stereo vision system of any of configurations (1) to        (10), wherein the automatic system calibration is comprised of a        calibration health indicator that indicates an accuracy of the        automatic system calibration.    -   (12) The stereo vision system of any of configurations (1) to        (11), wherein the stereo matching is comprised of a depth        estimate confidence score for each pixel of the plurality of        stereo images.    -   (13) The stereo vision system of any of configurations (1) to        (12), wherein the first and second camera sensors are mounted on        any of: a vehicle, a car, a truck, a traffic light, a lamp post,        left-side and right-side view mirrors of a vehicle,        respectively, a roof line of a vehicle, an upper-left-side and        an upper-right-side of a windshield, respectively, and left and        right head lamps, respectively.    -   (14) The stereo vision system of any of configurations (1) to        (13), wherein the cost function is based on a number of valid        pixels in a disparity map.    -   (15) The stereo vision system of any of configurations (1) to        (14), wherein the cost function is optimized based on intrinsic        camera parameters.    -   (16) The stereo vision system of an of configurations (1) to        (15), wherein the cost function is optimized based on extrinsic        camera parameters.    -   (17) The stereo vision system of any of configurations (1) to        (16), wherein the at least one processor is configured to        provide a calibration health indicator.    -   (18) The stereo vision system of any of configurations (1) to        (17), further comprising: a close-range sensor system configured        to provide 3D data for objects in a range that is closer than a        minimum depth-range of the first and second camera sensors,        wherein the at least one processor is configured to combine the        3D data produced from the first and second sensor signals with        the 3D data provided by the close-range sensor system.    -   (19) The stereo vision system of configuration (18), wherein the        close-range sensor system is comprised of a pair of camera        sensors having a wider field-of-view and a shorter baseline        distance than the first and second camera sensors.    -   (20) The stereo vision system of configuration (18), wherein the        close-range sensor system is comprised of a third camera sensor        that forms a trinocular stereo system with the first and second        camera sensors, such that the first and second camera sensors        have a shorter baseline length than the first and third camera        sensors.    -   (21) The stereo vision system of configuration (18), wherein the        close-range sensor system is comprised of a time-of-flight        camera.    -   (22) The stereo vision system of any of configurations (1) to        (21), further comprising an active illumination device        configured to emit visible or infrared radiation towards a field        of view of the first and second camera sensors.    -   (23) The stereo vision system of configuration (22), wherein the        active illumination device is configured to: alternate between        emitting radiation and not emitting radiation, and emit        radiation in synchronization with an exposure interval of the        first camera sensor and an exposure interval of the second        camera sensor.    -   (24) The stereo vision system of configuration (22), wherein the        active illumination component is comprised of any one or any        combination of: a vertical cavity surface emitting laser array,        a radiation lamp that emits in a visible spectrum range, and a        radiation lamp that emits in a near-infrared spectrum range.    -   (25). The stereo vision system of any of configurations (1) to        (24), wherein the at least one processor is configured to:        compute structure-from-motion data from the first sensor signal        and from the second sensor signal, and estimate, using the        structure-from-motion data, 3D positions of objects that are        closer than a minimum depth-range of the first and second camera        sensors.

Methods for detection of an object according to the technology describedherein may be include various processes. Example methods includecombinations of processes (26) and (27), as follows:

-   -   (26) A computer-implemented method performed by one or more        processors for detection of an object in an image, the method        comprising: identifying connected component regions on the        object by color; determining an aspect ratio of each of the        connected component regions; determining a distance between the        connected component regions; and identifying the object to be a        known object based on the aspect ratios and the distance between        the connected component regions, wherein by a presence, a        location, and a size of the known object is determined from the        image.    -   (27) The method of process (26), wherein: the identifying of the        connected component regions identifies the color to be red, and        the identifying of the object identifies the object to be a stop        sign.

Methods to calibrate a stereo vision system according to the technologydescribed herein may be include various processes. Example methodsinclude combinations of processes (28) through (33), as follows:

-   -   (28) A computer-implemented calibration method performed by one        or more processors to calibrate a stereo vision system that        includes a first camera sensor configured to sense first        reflected energy of a first image and to generate first sensor        signals based on the first reflected energy and a second camera        sensor configured to sense second reflected energy of a second        image and generate second sensor signals based on the second        reflected energy, the method comprising: generating stereo        images from the first and second sensor signals; rectifying the        stereo images using stereo calibration parameters, to produce        rectified stereo images; performing a stereo matching on the        rectified stereo images; and performing an automatic system        calibration using a result of the stereo matching, wherein the        automatic system calibration is based on minimizing a cost        function.    -   (29) The method of process (28), wherein the automatic system        calibration is comprised of any one or any combination of: a        fast calibration procedure configured to generate first stereo        calibration parameters to compensate for high-frequency        perturbations based on tracking image points corresponding to        features in the first and second images; a medium calibration        procedure configured to generate second stereo calibration        parameters to compensate for medium-frequency perturbations        based on optimizing a number of valid pixels for dense stereo        block matching as a function of a relative camera pitch and a        relative camera roll, and a slow calibration procedure        configured to generate third stereo calibration parameters to        compensate for low-frequency perturbations based on optimizing a        number of valid pixels for dense stereo block matching as a        function of a relative camera pitch, a relative camera roll, a        relative camera yaw, and a relative camera position.    -   (30) The method of process (28) or process (29), wherein the        automatic system calibration is comprised of an absolute range        calibration procedure configured to generate a stereo        calibration parameter for a relative camera yaw based on a        non-negative disparity constraint.    -   (31) The method of any one of processes (28) to (30), wherein        the automatic system calibration is comprised of an absolute        range calibration procedure configured to generate a stereo        calibration parameter for a relative camera yaw based on        dimensions of an object in a scene of a plurality of stereo        images and a focal length of the first and second camera        sensors.    -   (32) The method of any one of processes (28) to (31), wherein        the automatic system calibration is comprised of an absolute        range calibration procedure configured to generate a stereo        calibration parameter for a relative camera yaw based on vehicle        odometry.    -   (33) The method of any one of processes (28) to (32), wherein        the automatic system calibration is comprised of a calibration        management procedure configured to manage an operation schedule        of the slow calibration procedure, the medium calibration        procedure, the fast calibration procedure, and an absolute range        calibration procedure.

A non-transitory computer readable medium storing computer-executablecode to calibrate a stereo vision system according to the technologydescribed herein may be embodied in different configurations. Exampleconfigurations include combinations of configurations (34) through (39),as follows:

-   -   (34) A non-transitory computer readable medium storing        computer-executable code that, when executed by one or more        processors, causes the one or more processors to calibrate a        stereo vision system that includes a first camera sensor        configured to sense first reflected energy of a first image and        to generate first sensor signals based on the first reflected        energy and a second camera sensor configured to sense second        reflected energy of a second image and generate second sensor        signals based on the second reflected energy, wherein the method        is comprised of: generating stereo images from the first and        second sensor signals; rectifying the stereo images using stereo        calibration parameters, to produce rectified stereo images;        performing a stereo matching on the rectified stereo images; and        performing an automatic system calibration using a result of the        stereo matching, wherein the automatic system calibration is        based on minimizing a cost function.    -   (35) The non-transitory computer readable medium of        configuration (34), wherein the automatic system calibration is        comprised of any one or any combination of: a fast calibration        procedure configured to generate first stereo calibration        parameters to compensate for high-frequency perturbations based        on tracking image points corresponding to features in the first        and second images; a medium calibration procedure configured to        generate second stereo calibration parameters to compensate for        medium-frequency perturbations based on optimizing a number of        valid pixels for dense stereo block matching as a function of a        relative camera pitch and a relative camera roll, and a slow        calibration procedure configured to generate third stereo        calibration parameters to compensate for low-frequency        perturbations based on optimizing a number of valid pixels for        dense stereo block matching as a function of a relative camera        pitch, a relative camera roll, a relative camera yaw, and a        relative camera position.    -   (36) The non-transitory computer readable medium of        configuration (34) or configuration (35), wherein the automatic        system calibration is comprised of an absolute range calibration        procedure configured to generate a stereo calibration parameter        for a relative camera yaw based on a non-negative disparity        constraint.    -   (37) The non-transitory computer readable medium of any one of        configurations (34) to (36), wherein the automatic system        calibration is comprised of an absolute range calibration        procedure configured to generate a stereo calibration parameter        for a relative camera yaw based on dimensions of an object in a        scene of a plurality of stereo images and a focal length of the        first and second camera sensors.    -   (38) The non-transitory computer readable medium of any one of        configurations (34) to (37), wherein the automatic system        calibration is comprised of an absolute range calibration        procedure configured to generate a stereo calibration parameter        for a relative camera yaw based on vehicle odometry.    -   (39) The non-transitory computer readable medium of any one of        configurations (34) to (38), wherein the automatic system        calibration is comprised of a calibration management procedure        configured to manage an operation schedule of the slow        calibration procedure, the medium calibration procedure, the        fast calibration procedure, and an absolute range calibration        procedure.

It should be understood that the embodiments and examples describedherein have been chosen and described in order to illustrate theprinciples, methods, and processes of the inventive technology and itspractical applications to thereby enable one of ordinary skill in theart to utilize the inventive technology in various embodiments and withvarious modifications as are suited for particular uses contemplated.Even though specific embodiments of the inventive technology have beendescribed, they are not to be taken as exhaustive. Other embodiments andvariations that will be apparent to those skilled in the art but thatare not specifically described herein are within the scope of thepresent technology.

Unless stated otherwise, the terms “approximately” and “about” are usedto mean within ±20% of a target value in some embodiments, within ±10%of a target value in some embodiments, within ±5% of a target value insome embodiments, and yet within ±2% of a target value in someembodiments. The terms “approximately” and “about” can include thetarget value. The term “essentially” is used to mean within ±3% of atarget value.

The technology described herein may be embodied as a method, of which atleast some acts have been described. The acts performed as part of themethod may be ordered in any suitable way. Accordingly, embodiments maybe implemented in which acts are performed in an order different thandescribed, which may include performing some acts simultaneously, eventhough described as sequential acts in illustrative embodiments.Additionally, a method may include more acts than those described, insome embodiments, and fewer acts than those described in otherembodiments.

Various aspects of the present disclosure may be used alone, incombination, or in a variety of arrangements not specifically discussedin the embodiments described in the foregoing and is therefore notlimited in its application to the details and arrangement of componentsset forth in the foregoing description or illustrated in the drawings.For example, aspects described in one embodiment may be combined in anymanner with aspects described in other embodiments.

Use of ordinal terms such as “first,” “second,” “third,” etc., in theclaims to modify a claim element does not by itself connote anypriority, precedence, or order of one claim element over another or thetemporal order in which acts of a method are performed, but are usedmerely as labels to distinguish one claim element having a certain namefrom another element having a same name (but for use of the ordinalterm) to distinguish the claim elements.

Phraseology and terminology used herein is for the purpose ofdescription and should not be regarded as limiting. The use of“including,” “comprising,” or “having,” “containing,” “involving,” andvariations thereof herein, is meant to encompass the items listedthereafter and equivalents thereof as well as additional items.

The indefinite articles “a” and “an,” as used herein in thespecification and in the claims, unless clearly indicated to thecontrary, should be understood to mean “at least one.”

Any use of the phrase “at least one,” in reference to a list of one ormore elements, should be understood to mean at least one elementselected from any one or more of the elements in the list of elements,but not necessarily including at least one of each and every elementspecifically listed within the list of elements and not excluding anycombinations of elements in the list of elements. This definition alsoallows that elements may optionally be present other than the elementsspecifically identified within the list of elements to which the phrase“at least one” refers, whether related or unrelated to those elementsspecifically identified.

Any use of the phrase “equal” or “the same” in reference to two values(e.g., distances, widths, etc.) means that two values are the samewithin manufacturing tolerances. Thus, two values being equal, or thesame, may mean that the two values are different from one another by±5%.

The phrase “and/or,” as used herein in the specification and in theclaims, should be understood to mean “either or both” of the elements soconjoined, i.e., elements that are conjunctively present in some casesand disjunctively present in other cases. Multiple elements listed with“and/or” should be construed in the same fashion, i.e., “one or more” ofthe elements so conjoined. Other elements may optionally be presentother than the elements specifically identified by the “and/or” clause,whether related or unrelated to those elements specifically identified.Thus, as a non-limiting example, a reference to “A and/or B”, when usedin conjunction with open-ended language such as “comprising” can refer,in one embodiment, to A only (optionally including elements other thanB); in another embodiment, to B only (optionally including elementsother than A); in yet another embodiment, to both A and B (optionallyincluding other elements); etc.

As used herein in the specification and in the claims, “or” should beunderstood to have the same meaning as “and/or” as defined above. Forexample, when separating items in a list, “or” or “and/or” shall beinterpreted as being inclusive, i.e., the inclusion of at least one, butalso including more than one, of a number or list of elements, and,optionally, additional unlisted items. Only terms clearly indicated tothe contrary, such as “only one of” or “exactly one of,” or, when usedin the claims, “consisting of,” will refer to the inclusion of exactlyone element of a number or list of elements. In general, the term “or”as used herein shall only be interpreted as indicating exclusivealternatives (i.e. “one or the other but not both”) when preceded byterms of exclusivity, such as “either,” “one of,” “only one of,” or“exactly one of.” “Consisting essentially of,” if used in the claims,shall have its ordinary meaning as used in the field of patent law.

The term “substantially” if used herein may be construed to mean within95% of a target value in some embodiments, within 98% of a target valuein some embodiments, within 99% of a target value in some embodiments,and within 99.5% of a target value in some embodiments. In someembodiments, the term “substantially” may equal 100% of the targetvalue.

Also, some of the embodiments described above may be implemented as oneor more method(s), of which some examples have been provided. The actsperformed as part of the method(s) may be ordered in any suitable way.Accordingly, embodiments may be constructed in which acts are performedin an order different than illustrated or described herein, which mayinclude performing some acts simultaneously, even though shown assequential acts in illustrative embodiments.

Further, although advantages of the present invention may be indicated,it should be appreciated that not every embodiment of the invention willinclude every described advantage. Some embodiments may not implementany features described as advantageous herein. Accordingly, theforegoing description and attached drawings are by way of example only.

REFERENCES CITED

-   U.S. Pat. No. 6,392,688B1;-   U.S. Pat. No. 8,797,387B2;-   U.S. Ser. No. 10/097,812B2;-   JP2008-509619A; and-   Timo Hinzmann, Tim Taubner, and Roland Siegwart, “Flexible Stereo:    Constrained, Non-rigid, Wide-baseline Stereo Vision for Fixed-Wing    Aerial Platforms,” IEEE International Conference on Robotics and    Automation (ICRA), Brisbane AU (2018).

1.-39. (canceled)
 40. A stereo vision system, comprising: a first camerasensor configured to obtain a plurality of frames of a first image andto generate first sensor signals corresponding to the frames of thefirst image; a second camera sensor configured to obtain a plurality offrames of a second image and to generate second sensor signalscorresponding to the frames of the second image, the second camerasensor being untethered from the first camera sensor and separated fromthe first camera by at least 20 cm; at least one processor configured toreceive the first sensor signals from the first camera sensor and thesecond sensor signals from the second camera sensor in real time ornearly real time when the frames of the first and second images arebeing obtained, and to produce three-dimensional (3D) data from thefirst and second sensor signals, wherein the at least one processor isconfigured to: generate stereo images from the frames of the first imageand the frames of the second image, generate a depth map from the framesof the first image and the frames of the second image, and perform anautomatic system calibration based on the stereo images by minimizing acost function, wherein the automatic system calibration is comprised ofa stereo image stabilization performed at intervals of one frame each.41. The stereo vision system of claim 40, wherein a baseline distanceseparating the first and second camera sensors is in a range of 20 cm to1 m.
 42. The stereo vision system of claim 40, wherein a baselinedistance separating the first and second camera sensors is at least 1 m.43. The stereo vision system of claim 40, wherein a baseline distanceseparating the first and second camera sensors is in a range of 1 m to 5m.
 44. The stereo vision system of claim 40, wherein the first andsecond camera sensors are arranged to enable a depth range of about 30 mfrom the first and second camera sensors.
 45. The stereo vision systemof claim 40, further comprising: a third camera sensor configured toobtain a plurality of frames of a third image and to generate thirdsensor signals corresponding to the frames of the third image; and afourth camera sensor configured to obtain a plurality of frames of afourth image and to generate fourth sensor signals corresponding to theframes of the fourth image, wherein the least one processor isconfigured to: receive the third sensor signals from the third camerasensor and the fourth sensor signals from the fourth camera sensor inreal time or nearly real time when the frames of the third and fourthimages are being obtained, and to produce three-dimensional (3D) datafrom the third and fourth sensor signals, generate second stereo imagesfrom the frames of the third image and the frames of the fourth image,generate a depth map from the frames of the third image and the framesof the fourth image, and perform a second automatic system calibrationon the second stereo images based on minimizing a cost function, whereinthe second automatic system calibration is comprised of a stereo imagestabilization performed at intervals of one frame each.
 46. The stereovision system of claim 45, wherein: the depth map generated from theframes of the first image and the frames of the second image is a firstdepth map, the depth map generated from the frames of the third imageand the frames of the fourth image is a second depth map, and a depthrange of the first depth map is different from a depth range of thesecond depth map.
 47. The stereo vision system of claim 45, wherein: thefirst and second camera sensors are part of a long-range sensing system,the third and fourth camera sensors are part of a close-range sensingsystem, and the close-range sensing system is configured to detectobjects closer to the first and second camera sensors than a minimumdepth-range of the long-range sensing system.
 48. The stereo visionsystem of claim 40, wherein the first and second camera sensors areconfigured to be mounted on a vehicle.
 49. The stereo vision system ofclaim 48, wherein the first and second camera sensors are configured tobe mounted on left and right side-view mirrors, respectively, of thevehicle.
 50. The stereo vision system of claim 48, wherein the first andsecond camera sensors are configured to be mounted on a roof of thevehicle.
 51. The stereo vision system of claim 48, wherein the first andsecond camera sensors are configured to be mounted on a windshield ofthe vehicle.
 52. The stereo vision system of claim 48, wherein the firstand second camera sensors are configured to be mounted on left and rightheadlamp frames, respectively, of the vehicle.
 53. The stereo visionsystem of claim 40, further comprising: a third camera sensor configuredto obtain a plurality of frames of a third image and to generate thirdsensor signals corresponding to the frames of the third image, wherein:a distance separating the first and second camera sensors is a firstbaseline distance, and a distance separating the second and third camerasensors is a second baseline distance shorter than the first baselinedistance, and wherein the least one processor is configured to: receivethe third sensor signals from the third camera sensor in real time ornearly real time when the frames of the third image are being obtained,and to produce three-dimensional (3D) data from the second and thirdsensor signals, generate second stereo images from the frames of thesecond image and the frames of the third image, generate a depth mapfrom the frames of the second image and the frames of the third image,and perform a second automatic system calibration on the second stereoimages based on minimizing a cost function, wherein the second automaticsystem calibration is comprised of a stereo image stabilizationperformed at intervals of one frame each.
 54. The stereo vision systemof claim 53, wherein: the depth map generated from the frames of thefirst image and the frames of the second image is a first depth map, thedepth map generated from the frames of the second image and the framesof the third image is a second depth map, and a depth range of the firstdepth map is different from a depth range of the second depth map. 55.The stereo vision system of claim 54, wherein: the first and secondcamera sensors are part of a long-range sensing system, the second andthird camera sensors are part of a close-range sensing system, and theclose-range sensing system is configured to detect objects closer to thefirst and second camera sensors than a minimum depth-range of thelong-range sensing system.
 56. The stereo vision system of claim 40,wherein the automatic system calibration is comprised of: a fastcalibration procedure performed at fast intervals to compensate forhigh-frequency perturbations, a medium calibration procedure performedat medium intervals to compensate for medium-frequency perturbations,each medium interval being longer than each fast interval, and a slowcalibration procedure performed at slow intervals to compensate forlow-frequency perturbations, each slow interval being longer than eachmedium interval.
 57. The stereo vision system of claim 56, wherein: theautomatic system calibration is comprised of an absolute rangecalibration procedure performed at intervals longer than each fastinterval and shorter than each medium interval, and the absolute rangecalibration procedure is configured to generate a stereo calibrationparameter for a relative camera yaw based on dimensions of an object ina plurality of the stereo images and a focal length of the first andsecond camera sensors.
 58. A computer-implemented calibration methodperformed by one or more processors to calibrate a stereo vision systemthat includes a first camera sensor configured to obtain a plurality offrames of a first image and to generate first sensor signalscorresponding to the frames of the first image, a second camera sensorconfigured to obtain a plurality of frames of a second image and togenerate second sensor signals corresponding to the frames of the secondimage, a third camera sensor configured to obtain a plurality of framesof a third image and to generate third sensor signals corresponding tothe frames of the third image, and a fourth camera sensor configured toobtain a plurality of frames of a fourth image and to generate fourthsensor signals corresponding to the frames of the fourth image, themethod comprising: generating, by the one or more processors, firststereo images from the frames of the first image and the frames of thesecond image; generating, by the one or more processors, a first depthmap from the frames of the first image and the frames of the secondimage; generating, by the one or more processors, second stereo imagesfrom the frames of the third image and the frames of the fourth image;generating, by the one or more processors, a second depth map from theframes of the third image and the frames of the fourth image; andperforming, by the one or more processors, an automatic systemcalibration based on the first stereo images and the second stereoimages by minimizing a cost function, wherein the automatic systemcalibration is comprised of a stereo image stabilization performed atintervals of one frame each, and wherein a depth range of the firstdepth map is different from a depth range of the second depth map.
 59. Anon-transitory computer-readable medium storing computer-executable codethat, when executed by one or more processors, causes the one or moreprocessors to perform a method of calibrating a stereo vision systemthat includes a first camera sensor configured to obtain a plurality offrames of a first image and to generate first sensor signalscorresponding to the frames of the first image, a second camera sensorconfigured to obtain a plurality of frames of a second image and togenerate second sensor signals corresponding to the frames of the secondimage, a third camera sensor configured to obtain a plurality of framesof a third image and to generate third sensor signals corresponding tothe frames of the third image, and a fourth camera sensor configured toobtain a plurality of frames of a fourth image and to generate fourthsensor signals corresponding to the frames of the fourth image, whereinthe method is comprising of: generating first stereo images from theframes of the first image and the frames of the second image; generatinga first depth map from the frames of the first image and the frames ofthe second image; generating second stereo images from the frames of thethird image and the frames of the fourth image; generating a seconddepth map from the frames of the third image and the frames of thefourth image; and performing an automatic system calibration based onthe first stereo images and the second stereo images by minimizing acost function; wherein the automatic system calibration is comprised ofa stereo image stabilization performed at intervals of one frame each,and wherein a depth range of the first depth map is different from adepth range of the second depth map.